The Claims: 



1 . (previously amended) A system that enables a user to interact with a virtual control 
panel for controlling an industrial device or system, the system comprising: 

a pointing object carried and manipulated by the user for interacting with the virtual 
control panel to control the industrial device or system, 

a first tracking unit adapted to capture data representing a position of the pointing object, 

a portable identification element carried and manipulated by the user, the portable 
identification element being configured to define a position and orientation of the virtual control 
panel, 

a second tracking unit adapted to capture data representing a position of the portable 
identification element, 

a storage unit configured to store at least one pre-defined graphical interface representing 
a control panel of the industrial device or system, 

a graphics unit configured to generate a graphical representation of the control panel 
based on said stored graphical interface, 

a registering unit configured to register said graphical representation of the control panel 
in a fixed relation to said portable identification element, based on said data representing the 
position of the portable identification element, to produce the virtual control panel, 

a display unit configured to show the user a view comprising the real world and the 
virtual control panel projected in a fixed relation to said portable identification element, and 

an application unit configured communicate with the industrial device or system to 
perform actions to control the industrial device or system in response to the interactions of the 
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user with the virtual control panel, and configured to determine which actions to control the 
industrial device or system to perform based on the position of said user controlled pointing 
object in relation to the identification element. 

2. (previously amended) The system according to claim I, wherein the system is 
adapted to modify the appearance of the virtual control panel in response to interactions between 
the user controlled pointing object and the virtual control panel. 

3. (previously amended) The system according to claim 1, wherein said graphical 
interface is adapted to display data from the device and wherein the system is adapted to generate 
a graphical representation of the data and to display the data on the virtual control panel. 

4. (previously amended) The system according to claim 1, wherein said user controlled 
pointing object is a handheld pointing device or a part a body of the user. 

5. (previously amended) The system according to claim 1, wherein the storage unit is 
adapted to store a plurality of graphical interfaces, each representing a control panel of a 
particular device, wherein the system is adapted to generate and display a plurality of graphical 
representations of control panels for different devices based on said stored graphical interfaces of 
the devices, and wherein the system further comprises: 

an identification unit configured to identify which of the stored control panels to be 
displayed. 
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6. (previously amended) The system according to claim 5, wherein said identification 
unit comprises a recognition unit configured to recognize and identify devices in the 
environment of the user, and wherein the system is adapted to determine which of the stored 
control panels to be displayed based on which of the devices is identified. 

7. (previously amended) The system according to claim 6, wherein said recognition unit 
is adapted to recognize and identify unique identification markings on the devices. 

8. (previously amended) The system according to claim 5, wherein the system is 
arranged so the virtual control panel displayed changes when another device is recognized and 
identified, and when the user has accepted the other device. 

9. (previously amended) The system according to claim 1, wherein said portable 
identification element is adapted to be carried by the user during interaction with the virtual 
control panel. 

10. (previously amended) The system according to claim 1, wherein said portable 
identification element is attachable to a body of the user. 

1 1 . (previously amended) The system according to claim 1 , wherein said display unit 
comprises a wearable display device showing the user said view. 

12. (previously amended) A method that enables a user to interact with a virtual control 
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panel for controlling an industrial device or system using a user controlled pointing object, the 
method comprising: 

receiving data representing a position of the user controlled pointing object, 

receiving data representing a position of a portable identification element carried and 
manipulated by the user, 

storing at least one pre-defined graphical interface representing a control panel of the 
industrial device or system, 

generating a graphical representation of the control panel of the device based on said pre- 
defined graphical interface, 

registering said graphical representation of the control panel in a fixed relation to said 
portable identification element, based on said data representing the position of the identification 
element, to produce the virtual control panel in a position and orientation defined by the user 
with the portable identification unit, 

displaying a view comprising the real world and the virtual control panel projected in a 
fixed relation to said portable identification element, and 

performing actions to communicate with the industrial device or system to control the 
industrial device or system in response to the interactions of the user with the virtual control 
panel, wherein the actions to control the industrial device or system to be performed is 
determined based on the position of said user controlled pointing object in relation to the position 
of the virtual control panel. 

13. (previously amended) The method according to claim 12, further comprising: 
modifying the appearance of the virtual control panel in response to interactions between 
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the user controlled pointing object and the virtual control panel. 

14. (previously amended) The method according to claim 12, further comprising: 
defining a two-way communication between the virtual control panel and the device, 
sending information to the device regarding the users actions with the virtual control 

panel, 

receiving data from the device, generating a graphical representation of the received data 

and 

displaying the data on the virtual control panel. 

15. (previously amended) The method according to claim 14, wherein said data is 
displayed on the virtual control panel in response to interactions between the user controlled 
pointing object and the virtual control panel. 

16. (previously amended) The method according to claim 12, further comprising: 
storing a plurality of pre-defined graphical interfaces, each representing a control panel of 

a particular device, 

determining which of the stored control panels to be displayed, and 

generating a graphical representation of the control panel to be displayed based on the 

pre-defined graphical interface of the control panel to be displayed. 

17. (previously amended) The method according to claim 12, wherein at least one of the 
stored graphical interfaces comprises more than one graphical view to be displayed on the virtual 
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control panel, and which of the views to be displayed is determined based upon actions of the 
user. 

18. (previously amended) The method according to claim 16, further comprising: 
recognizing and identifying a device, 

determining which of the stored control panels to be displayed based on the identified 
device, and 

generating graphical representation of the control panel of the identified device based on 
the stored graphical interface of the identified device and displaying a view comprising the real 
world and the virtual control panel of the identified device projected in a fixed relation to said 
portable identification element. 

19. (previously amended) The method according to claim 18, wherein each device is 
provided with a unique identification marking and a device is recognized by identifying its the 
unique identification marking. 

20. (previously amended) The method according to claim 18, wherein the virtual control 
panel displayed is changed when another device is recognized and identified, and when the user 
has accepted the device. 

21. (previously amended) The method according to claim 12, wherein said portable 
identification element is carried by the user during interaction with the virtual control panel. 
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22. (previously amended) The method according to claim 12, wherein the virtual control 
panel comprises virtual interaction members and an audio and/or visual feedback is generated 
when the user activates any of the virtual interaction members. 

23. (previously amended) A computer program product, comprising: 
a non-transitory computer readable medium; and 

program instructions recorded on the computer readable medium which, when loaded 
into a computer, causes the computer to perform a method that enables a user to interact with a 
virtual control panel for controlling an industrial device or system using a user controlled 
pointing object, the method comprising 

receiving data representing a position of the user controlled pointing object, 

receiving data representing a position of a portable identification element carried and 
manipulated by the user, 

storing at least one pre-defined graphical interface representing a control panel of the 
industrial device or system, 

generating a graphical representation of the control panel of the device based on said pre- 
defined graphical interface, 

registering said graphical representation of the control panel in a fixed relation to said 
portable identification element, based on said data representing the position of the identification 
element, to produce the virtual control panel in a position and orientation defined by the user 
with the portable identification unit, 

displaying a view comprising the real world and the virtual control panel projected in a 
fixed relation to said portable identification element, and 
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performing actions to communicate with the industrial device or system to control the 
industrial device or system in response to the interactions of the user with the virtual control 
panel, wherein the actions to control the industrial device or system to be performed is 
determined based on the position of said user controlled pointing object in relation to the position 
of the virtual control panel. 

24. (cancelled) 
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